Deep Learning: How Stacked Simple Layers Learn to See

Point your phone at a street sign in another language and the words turn into your own, drawn over the live picture. Ask it to find "dog" in ten thousand photos and it does. Twenty years ago both would have been research projects with a few percent of the accuracy. Nobody wrote a rule for what a dog looks like. A network with tens of millions of adjustable numbers was shown labelled pictures and nudged until its guesses improved.

The previous course, Machine Learning (34/32), taught the loop: a model with knobs, a score for how wrong it is, and gradient descent to turn the knobs. Deep learning is that loop applied to a very particular kind of model, one made of many layers, and the engineering that makes the loop work when the model is deep, wide and hungry for data. A graduate keeps a short list of facts about why this works at all and what breaks when it does not.

The first widely noticed success was in 2012. A convolutional network called AlexNet, trained by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton on two gaming graphics cards, reached a 15.3% top-5 error rate on the ImageNet photo-recognition contest, more than ten percentage points better than the runner-up. In 2018 Yoshua Bengio, Geoffrey Hinton and Yann LeCun received the Turing Award for their work on deep networks, and in 2024 John Hopfield and Hinton shared the Nobel Prize in Physics for foundational work on artificial neural networks.

Depth: stacked bends that fold the input

A layer of a modern network does two things. It takes weighted sums of its inputs, then bends the result with a simple non-linear function. The most common bend is the ReLU: pass positive numbers through unchanged, replace negatives with zero. With ReLUs, a whole network is a patchwork of flat straight pieces joined at kinks.

One hidden layer is already enough, in principle, to approximate any smooth function: this is the universal approximation theorem (Cybenko and Hornik, both 1989). But the theorem says nothing about how wide that layer must be, and the answer can be absurd. Depth is the cheaper route. Each layer takes the output of the one below and bends it again, so the number of pieces can multiply from layer to layer instead of merely adding. Telgarsky (2016) gave a clean example, and you can build it here.

Each pass folds the interval in half, like folding a strip of paper. After k folds the curve has 2^k straight pieces, built from only 2k hidden units. A single hidden layer with 2k units can make at most 2k + 1 pieces on a line.

Without JavaScript: 3 stacked layers use 6 hidden units and make 8 straight pieces; one wide layer with 6 units makes at most 7. At 7 layers it is 128 against 15.

That is the intuition a graduate keeps. Depth lets a network reuse the same cheap building block at every scale, so a pattern like "an eye, in a face, in a crowd" can be built from parts of parts. Real networks do not fold exactly, and the theory of why deep nets are so effective in practice is still active research. But the exponential count of pieces is a real result (Montúfar and colleagues, 2014, proved bounds of this kind), and it is the first reason why people stacked layers instead of widening one.

Convolution: one small detector slid everywhere

A photo of 224 by 224 pixels in three colours has 150,528 numbers. A fully connected layer that maps them to just 1,000 units needs 150,528 × 1,000 = about 150 million weights. That is too many to learn, and it is also silly: a cat in the corner and a cat in the centre would need to be learned separately.

A convolutional layer makes two bets. Nearby pixels matter together, so each unit looks at a small patch, say 3 by 3. And a useful detector in one place is useful in every place, so the same small set of weights, a filter, slides across the whole image. That layer with 64 filters over a three-colour image has 3 × 3 × 3 × 64 = 1,728 weights, whatever the size of the picture.

Stacked layers build up features: the first finds edges, the next corners and textures, later ones parts of objects. Yann LeCun's group used this design on handwritten zip codes in 1989 and refined it into LeNet-5 by 1998. AlexNet in 2012 was the same idea with more data, more layers and graphics cards. Try the filters below on a small picture.

Without JavaScript: a vertical-edge filter responds on the left and right sides of a filled square and nowhere else. Slide the square one pixel and the responses slide with it.

The sliding gives a property worth naming: equivariance. Move the cat and the pattern of responses moves with it. The sharing is also why convolutional nets need far less data than a dense net on pictures. When a graduate looks at a new data type, they ask the same question: what is the symmetry here, and can one shared detector exploit it? Sound has time, graphs have neighbours, molecules have rotations.

Deep chains fade or explode, and skip connections fix it

Backpropagation (taught in Machine Learning, 34/32) sends the error signal backwards through the layers, and at every layer it is multiplied by that layer's weights and by the slope of its bend. Multiply thirty numbers together and the outcome depends strongly on whether each is a bit below one or a bit above. Below one, the signal reaching the early layers shrinks to nothing and those layers stop learning: the vanishing gradient, analysed in Sepp Hochreiter's 1991 diploma thesis. Above one, it blows up.

The classic bend, the sigmoid, makes this worse. Its slope is never more than ¼, so even with weights of the right size, thirty layers can multiply the signal by something like 0.25³⁰, about 10^-18. The remedies are one of the stories of the field, and a graduate knows them as a list:

• A bend with slope 1 on its active side (ReLU), widely adopted around 2010 to 2012.
• Careful starting weights, scaled by layer width: Glorot and Bengio (2010) for sigmoid-like bends and He and colleagues (2015) for ReLU.
• Normalisation layers, such as batch normalisation (Ioffe and Szegedy, 2015).
• Skip connections, the key idea of residual networks: each block outputs x + F(x), so the signal has a direct path to the input. ResNet (He, Zhang, Ren and Sun) trained a 152-layer network and won the 2015 ImageNet contest.

Without JavaScript: with sigmoid bends, scale 1 and 30 layers, the gradient reaching the first layer is about 10^-19 of the one at the output. With skip connections it stays near 1.

Notice what the skip does and does not do. At scale 1.4 with ReLU, the skip grows the signal instead: residual branches have to start small (practitioners scale or zero-initialise them) so that stacking a hundred of them stays tame. The lesson a graduate keeps is not a recipe but a diagnosis. When a deep network does not train, check how big the signal is at each layer, forwards and backwards.

Optimisers: momentum and Adam in a narrow valley

Plain gradient descent steps straight downhill. In a long narrow valley, downhill points mostly at the valley's nearest wall, so the steps zigzag across it while creeping along the floor. Raise the learning rate to hurry and the zigzag grows until it diverges. The valley below is the simplest case, a bowl 5 times narrower in one direction than the other (curvature 25 against 1). For this bowl, gradient descent is stable only for learning rates below 2/25 = 0.08, a number you can derive in one line.

Momentum (Polyak, 1964) lets the step remember its recent direction, like a ball with inertia: zigzags cancel, the steady push along the valley adds up. Adam (Kingma and Ba, 2014, published at ICLR 2015) also rescales each parameter's step by the typical size of its recent gradients, so steep and shallow directions get comparable steps. It is the default in much of deep learning, with variants such as AdamW, though careful momentum-based training still wins in some settings, and which is best is argued case by case.

Without JavaScript: at rate 0.03 and 60 steps, plain descent reaches a loss of about 0.21, momentum about 0.019 and Adam (rate 0.3) about 0.0024. At rate 0.09 plain descent diverges.

More knobs than data, and it still generalises

The old rule from statistics said that a model with more parameters than data points will memorise the data and fail on new data. Deep networks routinely break the rule. Zhang and colleagues (ICLR 2017) showed that standard image networks can fit images with randomly shuffled labels perfectly, so they certainly have the capacity to memorise, and yet the same networks generalise well on real labels. Belkin and colleagues (PNAS, 2019) described a "double descent" curve: test error can rise as the model nears the point where it just fits the training data, then fall again as it grows larger. Why the large regime works is still being explained; the working summary is that the training process itself prefers simpler fits among the many that exist.

Practice leans on a few restraints, all cheap to state: weight decay (a small penalty on large weights), early stopping (stop at the lowest validation error), dropout (randomly silence units during training, Srivastava and colleagues, 2014), and data augmentation (flip, crop and jitter the pictures). Try the first two on 12 noisy points from a sine wave.

Without JavaScript: with 50 hidden units and no decay the validation error is lowest (0.125) after about 450 steps and has risen to 0.181 by step 4,000 while training error keeps falling to 0.007. With a decay of 0.001 the final validation error is 0.131.

Two cautions. A held-out set is still the only honest score, and the same split reused to choose settings slowly stops being a test (Machine Learning, 34/32). And all of this assumes new data resembles the old; a network that has never seen a rainy night can still be confidently wrong in one.

Attention: why scores are divided by the square root of d

How Large Language Models Actually Work (29/02) explains what attention does: every token compares its query with the other tokens' keys and takes a weighted mix of their values. Here is a detail that shows how a deep-learning researcher thinks. The comparison is a dot product of two vectors of length d. If the entries are independent numbers of typical size 1, a dot product adds d such products, and its typical size grows like the square root of d.

Those scores then go through softmax, which turns them into weights that sum to 1. Softmax is hypersensitive to large inputs: when one score leads by a few units it takes nearly all the weight, and the gradient through it nearly vanishes, the same disease as the previous section. The transformer paper (Vaswani and colleagues, NeurIPS 2017) divides the scores by √d for this reason. The whole fix is one division.

Without JavaScript: at d = 64 the scores have a typical size near 8, and the largest of six weights averages about 0.89 without scaling and about 0.42 with it.

What a graduate just sees

• A network is a composition of simple bends; with ReLU it is a patchwork of straight pieces, and depth multiplies the number of pieces where width only adds.
• When the data has a symmetry (position, time, order), one shared detector beats a separate one for every location.
• Training a deep net is a signal-flow problem: if the gradient dies or explodes between layers, nothing else matters yet.
• Skip connections, sensible initialisation and normalisation exist to keep the signal about the same size at every depth.
• The optimiser matters most in badly shaped valleys; momentum and Adam are cures for zigzag and for mismatched scales.
• Memorising and generalising are different things, and the network that could memorise often does not. Only held-out data tells you which happened.
• Small details such as one division by √d can decide whether a design trains.
• Whether further scale is enough is argued: Richard Sutton's "The Bitter Lesson" (2019) says general methods plus computation have always won, while critics such as Gary Marcus (2018) argue deep learning alone will hit limits that need other ingredients.

Where it shows up next

• Data Ethics, Privacy & Fairness (36/37): these models inherit patterns in their training data, and large ones can memorise and leak what they were trained on.
• Computational Linguistics (36/30): language models are deep networks trained to predict the next piece of text, with attention at the core.
• Big Data & Data Engineering (36/36): training runs on thousands of chips, and the data pipelines feeding them are an engineering field of their own.
• In the world: speech recognition and translation, medical-image screening tools under regulatory review, recommendation feeds, protein-structure prediction and weather emulators.

If you want to go deeper

• 📘 Textbook. Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning (MIT Press, 2016), the standard reference, free to read at deeplearningbook.org; and Simon Prince, Understanding Deep Learning (MIT Press, 2023), more recent and clearly illustrated, with a free draft at udlbook.github.io.
• 📘 Hands-on text. Aston Zhang, Zachary Lipton, Mu Li and Alexander Smola, Dive into Deep Learning, a free online book with runnable code at d2l.ai.
• ▶️ Gentler. Grant Sanderson's Neural Networks series on the 3Blue1Brown channel (2017 onward), for the pictures behind backpropagation, and Andrej Karpathy's Neural Networks: Zero to Hero video lectures, which build a small network and a language model from scratch.
• 🌐 Free course. Stanford's CS231n: Deep Learning for Computer Vision, whose notes and past lecture recordings are public; or Practical Deep Learning for Coders from fast.ai, a code-first free course.